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Concentration to Zero Bit-Error Probability for Regular LDPC Codes 
on the Binary Symmetric Channel: Proof by Loop Calculus 

Marc Vuffray^ and Theodor Misiakiewicz^ 


Abstract —In this paper we consider regular low-density 
parity-check codes over a hinary-symmetric channel in the 
decoding regime. We prove that up to a certain noise threshold 
the hit-error prohahility of the hit-sampling decoder converges 
in mean to zero over the code ensemble and the channel 
realizations. To arrive at this result we show that the hit-error 
prohahility of the sampling decoder is equal to the derivative 
of a Bethe free entropy. The method that we developed is new 
and is based on convexity of the free entropy and loop calculus. 
Convexity is needed to exchange limit and derivative and the 
loop series enables us to express the difference between the bit¬ 
error probability and the Bethe free entropy. We control the 
loop series using combinatorial techniques and a first moment 
method. We stress that our method is versatile and we believe 
that it can be generalized for LDPC codes with general degree 
distributions and for asymmetric channels. 

I. Introduction 

In 1968 Gallager [I] introduced error-correcting codes 
based on low-density parity-check (LDPC) matrices. Since 
then LDPC codes have been proven to be of great practical 
and theoretical relevance. LDPC codes perform very well 
under iterative decoding on a broad class of symmetric 
memoryless channels (BMS) [2], [3] and provably achieve 
capacity on the binary erasure channel (BEC) [4]. Since 1996 
they have been integrated into many industrial standards from 
wireless communications to computer chips. 

An important performance measure of an LDPC code 
and its associated decoder is the bit-error probability. It 
is the fraction of bits that are on average incorrectly re¬ 
constructed. The bit-error probability of LDPC codes un¬ 
der belief-propagation (BP) decoding is well-understood on 
BMS channels using the method of density evolution [5]. 
However it is a more challenging task to control the bit-error 
probability of the bit maximum a posteriori (MAP) decoder. 

Lower and upper bounds on the noise threshold for van¬ 
ishing bit-MAP error probability have already been derived 
in Gallager’s thesis [1] for a class of BMS channels. These 
bounds have been improved and generalized for every BMS 
channels by Shamai and Sason [6]. 

In an attempt to locate exactly the noise threshold, most 
of the attention has been focused on the conditional entropy 
per bit. Its derivative with respect to the channel noise 
is the so-called generalized extrinsic information transfer 
(GEXIT) curve [7]. The GEXIT curve is proportional to the 
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“magnetization” or bit-error probability of the bit-sampling 
decodeiQ for the BEC and the binary-input additive white 
Gaussian-noise channel (BAWGNC) [8]. The magnetization 
is an upper-bound on the bit-MAP error probability. Hence 
for these two channels GEXIT curves and bit-MAP error 
probabilities vanish in the same noise regime. 

Surprisingly the conditional entropy and its derivative are 
related to the BP algorithm and its associated Bethe free 
entropy. It has been first proven in [9], [10] on the BEC 
channel that the conditional entropy is equal to an averaged 
form of the Bethe free entropy over the code ensemble. 
Bounds between the averaged Bethe free entropy and the 
conditional entropy are derived in [11], [12], [13] based on 
the interpolation method of Guerra and Toninelli [14], [15]. 
Equality has been proven on the binary-symmetric channel 
(BSC) using cluster expansions in a low-noise regime [16] 
and in a high-noise regime [17]. More recently equality 
between the GEXIT curve and the derivative of the average 
Bethe free entropy has been generalized to all BMS chan¬ 
nels [18] combining the interpolation method and spatially- 
coupled codes [19]. 

Although the conditional entropy and its threshold are 
completely characterized for BMS channels, its exact relation 
to the bit-MAP error probability remains unclear in general 
and in particular for the BSC channel. Due to Eano’s inequal¬ 
ity the conditional entropy is always a lower-bound on the 
bit-error probability. However inspired by previous results 
this inequality is conjectured to be tight for LDPC codes on 
a wide class of channels. 

In this paper we prove that for regular LDPC codes over 
a BSC channel the “magnetization” or bit-error probability 
of the bit-sampling decoder vanishes up to a certain thresh¬ 
old. This result also shows that the posterior measure of 
LDPC codes concentrates over the LDPC ensemble and the 
noise realizations. To achieve this result we show that the 
magnetization is asymptotically equal to a perturbed version 
of the Bethe free entropy. The technique that we present is 
new and is based on loop calculus or loop series derived by 
Chertkov and Chernyak [20]. The loop series expresses the 
difference between a quantity and its Bethe counterpart as a 
sum over subgraphs. Proving that the loop series vanishes is 
tantamount to controlling a purely combinatorial object that 
depends solely on the LDPC graph ensemble. Suboptimal 
bounds on this object are obtained using McKay’s estimates 
[21] following an idea developed in [17], [22]. 

*The bit-sampling decoder assigns random values to decoded bits based 
on their posterior marginal distribution. 


The technique that we present has the advantage to be sim¬ 
ple and versatile. To emphasize this point we also show that 
our results can be easily transposed to the BEC. Moreover 
we stress that our proofs do not rely explicitly on properties 
of the channel. Hence, we believe that this technique can 
be use to analyze LDPC codes over channels that are not 
symmetric. 

In Section |II] we give a precise definition of the bit¬ 
sampling decoder and its associated bit-error probability and 
we present our main theorems. In Section |III] we derive the 
relation between the Bethe free entropy and the bit-error 
probability and we express the difference using loop calculus. 
In Section|III]we reduce the loop series to a counting problem 
that we control with a first moment method. Finally we 
discuss about future works and possible improvements in 
Section [V] 


II. Main Results 

A. Regular LDPC codes on BMS channels 


s is transmitted reads 
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where the normalization factor Z (r,s) in Equation Q is 
the partition function 
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B. Concentration of the Bit-Error Probability for the Sam¬ 
pling Decoder 

We are interested in the performance of regular LDPC 
codes with respect to the average bit-error probability of 
decoding. We consider the bit-sampling decoder 

g-sampimg according to ^ pi 2 .\ s), (5) 
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LDPC codes are defined by a regular bipartite graph E = 
{V, C, E) where V is the set of variable nodes, C is the set 
of check nodes and E = V x C is the set of undirected 
edges. There are n = \V\ variable nodes and m = |C| check 
nodes. 

We consider regular LDPC codes with variable-node de¬ 
grees I > 3 and check-node degrees r > 1. The design rate 
of the code is by definition i?des = 1 — l/r. 

An LDPC code is generated randomly. The graph L is 
drawn uniformly at random from the ensemble of {l,r) 
regular bipartite graphs. Throughout the paper we write Er [•] 
the expectation with respect to the ensemble of regular {I, r) 
bipartite graphs with uniform probability. 

Denote the neighbors of a variable node i G V (resp. a 
check node a G C) hy di = {a G C \ {i,a) G E} (resm 

hy da = {i GV \ {i,a) G E}). A codeword is a sequencq^ 

CT = S {—1,1}” that satisfies the parity-check sum 

Y[a, = i, (1) 

iGda 

for all check nodes a G C. 

We transmit a codeword with uniform prior over a BMS 
channel with transition probability q {si \ ai), where the 
output of the channel could take any real value Si G K. 
The symmetry property of the channel is expressed through 
the simple relation 

q (s* I (Ti) = q (si I -(Ji) . (2) 

We assume without loss of generality that the all-zero 

codeworcd is transmitted. Hence the output of the channel 
s = G K” is i.i.d. with distribution q {si \ -1-1). The 

posterior probability that the codeword ct is sent given that 

^As we use concepts from statistical physics it is more convenient to 
employ the binary alphabet { — 1,1} instead of the traditional {0, Ij. 

^In the binary alphabet {—1,1}, the all-zero codeword is the sequence 

{!,...,!}. 


where g_\(Ji denotes the sequence of variables g_ with the 
component removed. 

The bit-error probability of the bit-sampling decoder 
pbit-sampiing (jj^gctly related to the marginals of the posterior 
probability Q 
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where E^ [•] denotes the expectation with respect to the 
channel output distribution and (•)j^ denotes the average with 
respect to the posterior probability Q. The expected quan¬ 
tity in Equation (|6]l is sometimes referred as the averaged 
magnetization in the physics community. 

An important question is to know when the bit-error 
probability is vanishing in the limit where the codeword 
length goes to infinity. In this paper we consider two families 
of symmetric channels, the BEC and the BSC. The BEC has 
an output alphabet Si G (—1,0,1} and is characterized by 
transition probabilities 


qBEC (1 I 1) = 1 _ gBEC (0 I 1) ^ ^BEC I ^ 0^ 

(7) 

where e G [0,1] is the erasure probability. The BSC has 
binary outputs Si G {—1,1} and is characterized by the 
transition probabilities 


q^^^(lll) = l-p,q^^^(-lll)=p, (8) 


where p G [0,1/2] is the flipping probability. 

Before we state our theorems we need to introduce the 
domain 
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We also need to introduce the auxiliary function f : D [p) x 
[0,1] —> K. defined as follow^ 

f {xo,Xc,y,p) = -lh 2 {il-p)xo +pXc) 

+ (1 - p) /l 2 (xo) + ph 2 (Xc) 
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and the function k : [0,1]^ —>■ R that reads 

k{xo,Xc,p,p) = {pxc - (1 - p) So) 111 • (11) 

Our main contribution are the two theorems stated below 
which give sufficient conditions on the channel parameters 
for concentration of the bit-error probability of the sampling 
decoder. 


Theorem 1 (Concentration of the Bit-Error Probability for 
the BEC). Consider the ensemble of {l,r) regular LDPC 
codes on a BEC with erasure probability e. If the following 
function achieves its maximum only at the point 

argmax / (O, e) = {(0,0,0)} , 
(o,Xo,y)er>(e) 

then the bit-error probability of the sampling decoder con¬ 
verges in mean to zero in the large codeword limit 

pbit-sampling _ q 

The same theorem holds for the BSC with a similar 
condition. 


lim Er,s 


Theorem 2 (Concentration of the Bit-Error Probability for 
the BSC). Consider the ensemble of {l,r) regular LDPC 
codes on a BSC with flipping probability p. If the following 
function achieves its maximum only at the point 


argmax / {xQ,Xc,y,p) -C k {xo,Xc,p,p) = {(0,0,0)} , 

(xo.Xo,y)eD(p) 


then the bit-error probability of the sampling decoder con¬ 
verges in mean to zero in the large codeword limit 


lim Er,s 
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Remark 3. Knowing that vanishes implies that 

with high probability the posterior measure Q concentrates 
on configurations that are at a Hamming distance o (n) from 
the all-zero codeword. 

We perform the global optimization numerically and we 
find for a few cases the maximum value of noise eioop and 
Pioop for which Theorem [1] and Theorem hold. Critical 


'^The binaiy entropy h 2 (p) := — (!— p) In (1 — p)—plnp is computed 
in nat. 


values of noise are displayed in Table U for the BEC and in 
Table HI] for the BSC. 
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Thresholds for some regular LDPC code ensembles over the 

BEC WITH ERASURE PROBABILITY e. THE BELIEF-PROPAGATION 
THRESHOLD IS EBP, THE MAXIMUM A POSTERIORI THRESHOLD IS CmAP, 

THE Shannon threshold is esh and our threshold is eloop. 
Values of BP and MAP thresholds are erom [23]. 
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Thresholds for some regular LDPC code ensembles over the 

BSC WITH erasure PROBABILITYp. THE BELIEF-PROPAGATION 
THRESHOLD IS psp, THE MAXIMUM A POSTERIORI THRESHOLD IS PMAP. 
THE Shannon threshold is psh and our threshold is ploop- 
Values of BP and MAP thresholds are erom [23]. 


We would expect that the probability of error vanishes 
for e < emap and p < pmap- Although the thresholds that 
we found are reasonably close to emap and pmap for graphs 
with small degrees, they become worse in the limit of large 
degrees. A quick inspection of (fTOl i and (fTTl i shows that 
the functions f /I and k/l become independent of the noise 
parameter in the limit where I and r go to infinity with a 
fixed ratio l/r. It implies that pioop and Eioop vanish. This 
behavior is in the opposite direction to what we can expect 
as in the limit of large degrees pmap — Psh- In Section |V| 
we discuss about possible improvements in our analysis in 
order to make our thresholds tight. 

The rest of the paper is organized as follows. In SectionlTm 
we show that the bit-error probability is related to the 
derivative of the so-called free entropy. Using the loop series, 
we express the free entropy as a combinatorial sum over 
subgraphs. In Section |IV] we control the loop series with 
asymptotic estimates on subgraphs and Laplace’s method. 
We prove Theorems [T] and |2] in this section. In Section IV] we 
discuss future directions and ways to improve and generalize 
our results. 





















III. Free Entropy, Bethe Approximation and Loop 
Series 


A. The Free Entropy and its Relation to the Bit-Error Prob¬ 
ability 

The bit-error probability (| 6 ]l is related to a “perturbed” 
version of the partition function (01). Let ry G M be the 
perturbation parameter entering in the perturbed partition 
function 

^(r,s,p) := 

CT aGC \ i^da / 

( 12 ) 

iev 


Note that Z (T,s,r]) is a non-increasing function of p and 
Z (r, s, 0) is the original partition function (01. 

The free entropy is the (normalized) logarithm of the 
partition function (fT2l i 

:=-lnZ(r,s,? 7 ). (13) 

n 

A direct computation shows that the derivative of the free 
entropy with respect to its perturbation parameter reads 


drj 
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Therefore the bit-error probability is related to the average 
entropy through the following relation 

_ _ 2 pbit-sampling 

rj—0 

Since Z (L, s, rj) is a non-increasing function of rj, the free 
entropy is non-increasing as well. Moreover the free entropy 
is a convex function of 77 as it can easily be verified by 
taking twice the derivative with respect to rj. It implies that 
in order to show concentration of the bit-error probability it 
is sufficient to prove that there exists rj < 0 independent of n 
such that Er.s [4> (r, s, rj)] —)> 0. If this condition is true then, 
thanks to monotonicity, the limit is also equal to zero for all 
p G [rj,oo [. Linally convexity of the free entropy enables us 
to exchange limit and derivative (see [24, p. 203]). 

In order to prove that the free entropy vanishes we 
decompose it into two contributions; the Bethe free entropy 
that can be computed explicitly and the so-called loop series 
that is a sum over subgraphs of L. Using a first moment 
method and combinatorial tools from graph theory, we show 
that with high probability the loop series vanishes in the 
large codeword limit. The last statement implies that the free 
entropy is equal to the Bethe free entropy. 
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B. The Bethe Approximation 

The Bethe free entropy is an approximation of the free 
entropy ([T3l l. It is defined as a functional over “messages” 
that are probability distributions i/i^a (^i), {<^i) asso¬ 

ciated with the directed edges i ^ a, a ^ i of the graph. 
The messages satisfy the so-called belief-propagation (BP) 


equations. Lor the free entropy (fTST l the BP equations take 
the following form 

E n 
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where the symbol oc denotes equality up to a normalization 
factor and g_Q^ := {aj \ j G da}. 

The Bethe free entropy evaluated at a fixed point of the 
BP equations is a sum of local contributions from nodes and 
edges of the graph L = (U, C, E) 
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Note that once a fixed-point of the BP equations (ITbl l is 
found, computing the Bethe free entropy ( [TTl i is a computa¬ 
tionally easy task. 


C. Corrections to the Bethe Free Entropy: the Loop Series 

The difference between the free entropy and the Bethe 
free entropy can be expressed with the so-called loop series 
derived by Chertkov and Chernyak [20]. It takes the form 
of the logarithm of a weighted sum over subgraphs of L. 
These subgraphs are called “loops” for they have no dangling 
edges. Note that if L is a tree no such subgraph exists and 
we recover the well-known result that the Bethe free entropy 
is exact on trees. 

We recall that a subgraph g = (Vg,Cg, Eg) of T = 
(y, C, E) is any graph with vertex set Vg C V, factor 
node set Cg C C and edge set Eg C {Vg x Cg) H E. Lor 
simplicity we denote the relation “g is a subgraph of L” with 
the inclusion symbol p C L. We also denote the induced 
neighborhood in p of a variable node i G Vg (resp. check 
node a G Cg) by dgi = didVg (resp. by dga = dad Cg). 

The set of “loops” consists of any non-empty subgraphs, 
not necessarily connected, with no degree one variable-node 
and no degree one check-node 

£r := {5 C r I Vi G Vg, \dgi\ > 2 and Va G Cg, \dga\ > 2}. 

(19) 

The difference between the free entropy and the Bethe free 
entropy is related to the loop series through the following 




equation 

0 (r, 1, v) - (r, s,v) = I In , (20) 

where the argument of the logarithm is a weighted sum over 
loops 

^&V=1+ E (21) 

9G£r 

The weight function over loops depends on the BP fixed 
point at which the Bethe free entropy is evaluated and can 
be expressed as a product over the nodes inside a loop 


^(iL-2) id) ■= n n 
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The factors Ki and Ka entering in (l22l i depend only on 
messages that are associated with edges neighboring the 
nodes i &Vg and a G Cg 
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For a complete derivation of the loop series for graphical 
models associated with linear codes, we refer the reader to 
[25], 


D. The Decoding Regime and its BP Fixed-Point 

Note that the loop series, as well as the Bethe free 
entropy, are functions of fixed-points of the BP equations 
(fThl l. The fixed-point associated with the decoding regime is 
the ferromagnetic fixed-point 


icr^) = z/ka i<^^) = —(25) 

One can easily see that ferromagnetic messages (l25T l satisfy 
the BP equations (fThl ) regardless of the channel considered 
and of the value of the perturbation parameter p £ R. The 
ferromagnetic fixed-point (l25l l describes a state for which 
the most likely configuration is the all-zero codeword i.e. 
ai = -fl. This is the reason why this fixed-point is associated 


with the decoding regime. 

The Bethe free entropy (fTTl i evaluated at the ferromagnetic 
fixed-point simply reads 

^Bethe = i ^ In (g (si I -hi)) . (26) 

i^V 

The factors entering in the weight function (|2^ are computed 
using Equations (l24l i for check nodes 
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and Equation (|2^ for variable nodes 
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where in the last expression we have used the half log- 
likelihood variables 



qjsi I +1) 
qis^ I -1)' 


(29) 


Based on the expression of the factors (l27l) and (l28l l. the only 
subgraphs with a non-zero weight are those with an induced 
variable-node degree equal to I and even induced check-node 
degree. This motivates the definition of the ferromagnetic 
loops ensemble 


£p = {g G Cr I Vi, a £ p, \dgi\ = I and \dga\ is even}. 

(30) 

A loop that is not an element of the ferromagnetic ensemble 
has a zero weight. Moreover the weight of a ferromagnetic 
loop is always non-negative 


K+{g) = exp -2r;kg|-2VA(sO >0.(31) 




In order to see that K+ (g) is non-negative, notice that a 
sign is only associated with the factors Ki and is equal to 
(—1)^ Therefore a loop can only have a negative weight 
if the product l\Vg\ is odd. Note that this product is the 
number of edges in a loop counted from the variable-node 
perspective. Therefore it should be equal to the number of 
edges counted from the check-node perspective 


l\Vg\=J2\dga\. (32) 

0,^Cg 

Since for a ferromagnetic loop \dga\ is always even, I \Vg\ is 
also even and the weight of a loop is always non-negative. 

Using Equations (l20l i and (l26l l we can express the average 
free entropy (fT3l l in the simple form 
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Note that Equation (l33l l is valid for all BMS channels 
regardless of the noise parameter. However we can only 
expect that the ferromagnetic loop-series vanishes in the 
decoding regime. 


The weight OTI) of a loop g S H (xq, Xc, y) is only a 
function of its type {g) = A"+ (a:o, Xc)- Using the specific 
expression of the half log-likelihood ( l29l l for each channels 
we find the explicit form of the weight function for the BEC 


IV. Eirst Moment Method on the Loop Series 

We use a first moment method to prove that the ferromag¬ 
netic loop-series in Equation (l3^ vanishes. In our case it is 
based on Jensen’s inequality and consists of permuting the 
expectation over the graph ensemble and the logarithm in 
Equation (l33]) . 

Note that we cannot permute the expectation over the 
channel output realizations and the logarithm. It is easy to 
see that over the channel output realizations a loop has an 
expected weight (ijTl i that increases exponentially fast for 

p < 0 

¥.^[K+[g)]=e-'^\^o\. (34) 

This is because the loop series is dominated by events for 
which most of the bits are corrupted and have negative half 
log-likelihood (|29] |. These events are rare but give rise to an 
exponentially large weight. 

Therefore we estimate the expectation of the loop series 
over the ensemble of regular (Z, r) bipartite graphs for a fixed 
output realization of the channel. 


A. Probability Estimates on Graphs 


Eor a given channel realization s of the BEC (resp. BSC) 
call Vc the set of variable nodes with Si = 0 (resp. = — 1) 
and call Vq the set of variable nodes i G V with Si = 1 (resp. 
Si = 1). The set Vq contains bits that have been correctly 
transmitted and I4 contains bits that have been corrupted. We 
denote the fraction of correctly transmitted bits by (1 — p) = 
I Vb I /n and we denote the fraction of corrupted bits by p = 
1 14 1 /n. We recall that the total number of variable nodes is 
n = \V\ and the total number of check nodes is m = \C\. 

We decompose the set of ferromagnetic loops (l30l l into 
subsets of loops having the same “type”. The type of a 
loop g G is the triplet {x[),Xc,y) G where 

a^o = lUoflVbl/n is the fraction of correctly transmitted 
variable nodes in the loop, Xc= |I4 H I4I /n is the fraction 
of corrupted variable nodes in the loop and y = is 

the fraction of check nodes with degree 2t. The set of loops 
of type (^xo,Xc,y) is denoted by O (^xo,Xc,y)- 

Not all value of [xq, Xc, y) are admissible loop types. The 
fraction of check nodes inside a loop is upper bounded by 1. 
Moreover counting edges from the variable-node perspective 
or from the check-node perspective obviously gives the same 
number. Therefore types that are admissible belong to the 
following set already introduced in Section |II] Eq. @ 
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= *» = “, (36) 

10 xo > 0 

and for the BSC 

K^^‘^{xq,Xc) = exp {-2nr] {xo {1 - p) + Xcp) 

+nk(xo,Xc,p,p)) , (37) 

where k {xq, Xc, p,p) is the auxiliary function introduced in 
Section HI] Eq. (fTTT i 

k{xo,Xc,p,p) = {pxc- (1 - p) a:o) In . (38) 

Therefore the expected value of the loop series over the graph 
ensemble can be expressed only through loop types 
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(xo,Xe,y)eD(p) 

xEr [|U (a:o,a::c,y)|] . (39) 


The expected number of loops with prescribed type 
is upper bounded using McKay’s combinatorial 
estimat^ [21] for subgraphs with specified degrees 


Er[|U(a3o,Xe,y)|] < (E p) + ^,p) ) 


f n{l-p) np \ 
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where 5 i^r is a constant that depends only on I and r. 
McKay’s estimate has the advantage to have an asymptot¬ 
ically tight growth rate when n goes to infinity. 

It remains to prove that the average loop series ( |39] | with 
the bound (l40l i vanishes in the large n limit. 

B. Laplace’s Method and Proof of Theorems 

The loop series (l39l l is dominated by loop types that 
contribute to the sum with the biggest exponential growth. 
We apply Laplace’s method in order to characterize the 
biggest exponent. 

Using Stirling inequalities 

1 n! 1 

< , -<ei2n^ (41) 

V27me“"n" 

^McKay’s bound in its original form is only applicable for subgraphs of 
size less than n — . We refer to [17] for a careful analysis. 







we find an asymptotically tight upper bound on the estimate 
gOli 

Er (a;o,a;c,y)|] < exp (nf (xq, y, p)) , 

(42) 

where Ci^r and 5[ ^ are just numerical constants and 
/ (xq, Xc, y, p) is the auxiliary function introduced in Section 
HD Eq. 0 “ 

f{xo,Xc,y,p) = -//l 2 ((1 - p)xo + pXc) 

+ (1 - p) /i 2 (a;o) + ph 2 (xc) 
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Combining Equations ( l36] t, dJTl t and (l42l t, we show that the 
leading exponent in Equation < l39l t is for the EEC 

(p, v) = max / (O, Xc,y, p) - 2pXcP, (44) 

{0,Xa,v)GD{p) 

and is for the BSC 


Proof of Theorem [7] Let e be the probability of error of the 
BEC. First notice that the perturbed partition function (fTTI) is 
trivially lower bounded by 1 and upper bounded by . 

This implies that the free entropy ( fT3l l remains finite 


0 < ^ (E, s, p) < In 2 + 2 IpI . (46) 


Therefore using Equation (l33l l and the fact that (p) > 0 
we see that the loop series remains finite as well 


2 (ln 2 + |? 7 |) > |Es [(/i(r,s, 77 )] 

- J dsq{s I l)ln(p(s | 1 )) 


= E,, 


-In I 1+ E ^+(5) 
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(47) 


Let A be the following probabilistic event on the channel 
output realizations 
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(48) 


Output realizations in A are close to the average output 
realization. 


(p, p) = max {-2p (xq (1 - p) + Xcp) 

[xo,Xc,y)eD{p) 

+ / {xo,Xc,y,p) + fc(xo,Xc,p,p)) .(45) 

Notice that for all p and p the exponent q;®ec/bsc 
is non-negative. This is easily verified by evaluating the 
objective function at (xo,Xc,y) = (0,0,0). Therefore the 
bit-error probability vanishes if q^bec/bsc jg equal to 

zero for all 77 in a neighborhood of zero. The next Lemma 
shows that in fact only the maximization at p = 0 is 
important. 


Using Hoeffding’s inequality, we see that the probability 
of the complementary event A'^ vanishes 

Fs[A“]<^. (49) 

n~‘‘ 

Combining Jensen’s inequality and the trivial bound Wt\ on 
the loop series we have the following estimate 


Er,s 


- In I 1 + E (5) 


< ^ (In 2 -P IpI) 


Lemma 4. If the maximum of (1441) (resp. ( 1451 )) is uniquely 
achieved in (xq, Xc, y) = (0,0,0) for p = 0, then there exists 
p < 0 such that (p, p) = 0 (resp. (p^rj) = 0) for 
all p £ ]p, oo[. 

Proof See Appendix U □ 

In order to prove Theorems [T] and |2] we need to show 
that small variations around p do not change (p, 0 ) and 
q,bsc 'j’jjjg jg guaranteed by the following Lemma. 


+Es 

— In f 1 + Er 

E (5) 
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Since we have conditioned over channel output realizations 
that are in A, the fraction of corrupted bit is |p — e| < 
i/lnAi/n. Therefore combining Equation (02]), Lemma 0| 
and Lemma |5] we have that if (e^ 0) = 0 is uniquely 
achieved in (xo,Xc,y) = (0,0,0) then for all p G ]p,oo[ 
and n sufficiently large. 


Lemma 5. For all p G [0,1], if a®^‘'(p,0) = 0 (resp. 
q,bsc (p 0 ) = 0) and the maximum of (1441) (resp. (145b ) is 
uniquely achieved at (xo,Xc,y) = (0,0,0), there exists N 
sufficiently large such that 


Vn > N,\/S G 



qBEC/BSC 0) = 0 
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E 
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-ln(l-|-C 3 n^'‘), (51) 

n 

where C 3 and C 4 are numerical constants independent of n. 


Proof. See Appendix HU 


□ We have proved that for all p G ]p, cxd[ with p < 0 
the average free entropy converges in expectation over the 
regular (Z, r) LDPC ensemble 


We are now in position to prove our main theorems. 























lim Er 

n—¥co 


Es ['/'(rjS,??)] - J dsq{s I l)\n {q{s \ 1)) 


= 0 . 
^(52) 

In particular it implies that the average free entropy over the 
LDPC ensemble converges 

lim Er,s [(j) (F, s, rj)] = [ dsg (s | 1) In (q (s | 1)). (53) 

n—^oc ~ J 

Since Ep.s [<?!> (F, s, p)] is a convex function of rj and con¬ 
verges pointwise in a neighborhood of zero, we can exchange 
the limit and the derivative 
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® Er,s [</>(F,s,p)] 
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= lim ■^Er,s[(l){T,s,r])] 

n^oo or] ~ 
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= lim Er 

n—foo 


—E, [(^ (F,s,?7 )] 


77 = 0 


= —2 lim Er 

n—^oc 
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,bit-sampling 


(54) 


where in the last line we use Equation ( fTSl l that relates the 
free entropy to the bit-error probability. □ 

Theorem |2] has a proof almost identical to that of Theo¬ 
rem [T] 


V. Path Forward 

We would like to stress that the techniques developed in 
this paper are quite general. In particular they do not rely on a 
special form of channels or on the regular-degree distribution 
of the LDPC ensemble. Therefore we plan to improve our 
results in the following ways. 

A. Generalization to Arbitrary Degree Distributions 

The entire analysis can easily be extended to general 
degree distributions with bounded degrees. It will simply 
transform the function (03) that counts subgraphs into a more 
convoluted object. However extending our results to distri¬ 
butions with unbounded degrees, like for instance Poisson 
distributions, may be more complicated. One would have to 
derive an estimate for counting subgraphs in this particular 
case. 


B. Asymmetric Channels 

The loop series and the Bethe free entropy for general 
channels are almost exactly similar than for symmetric 
channels. For general channels we can no longer assume 
that the all-zero codeword is transmitted. Instead we have 
to average the bit-error probability over all possible input 
codewords t. In this case the weight of a loop remains similar 
than for symmetric channels. The weight is also non-negative 
and depends on the generalized half log-likelihood ratio 


A (Sj I Ti) 



g (g» I n) 

q {si I -Ti) ’ 


(55) 


where s denotes as usual the channel observations. In order to 
control the loop series, we will need to perform a conditioned 


expectation in (fSOl l over joint typical sequences of input 
codewords and noise realizations. 

C. Tight Thresholds 

As described in Section El the thresholds that we obtain 
are not tight. In fact at fixed rate they become worse and 
converge to zero as the degrees of the graph become large. 
The reason why we obtain such loose bounds for large 
degrees comes from the function / (^xo,Xc,y, p) defined in 
(I43I 1. This function counts the growing rate of the average 
number of subgraphs with a prescribed type (^xo,Xc,y) 

f= lim -ln(Er [|H (xo,a:c,y)|]) ■ (56) 

n^oo ri LI \ _/ ij / 

One can verify that if instead of / we use the function 

/= lim -Er [in (|0 (a;o,a;c,t/)|)] , (57) 

we obtain tight lower and upper bound on the threshold for 
vanishing bit-error probability. 

The function / only depends on the random graph ensem¬ 
bles that we consider and does not depend on a particular 
channel. Computing this function would provide a proof of 
the exact location of the MAP threshold for an extensive 
class of channels. However this computation could prove to 
be a very difficult task. 

A way around the problem of computing dig is to 
condition the expectation (l56l l on some rare events with 
respect to the random graph measure. Note that by Jensen’s 
inequality / is always upper-bounded by /. This is because 
the expectation (l5^ is dominated by rare events that are 
associated with a large weight |H (xq, Xc, y) |. Conditioning 
on these rare events will lead to better estimates of dig 
and will provide tighter bounds at least in the limit of large 
degrees. 


Appendix I 
Proof of Lemma 0 ] 

Proof. We prove Lemma |4] only for the BSC (the proof for 
the BEC is almost identical). For a given p and p, let us 
define the following function 

(xo,Xc,y,?7) = f {xo,Xc,y,p) + k{xQ,Xc,p,p) 

-2r] (xo {1- p)+ Xcp) ■ (58) 

The function (xq, Xc, y, p) corresponds to the expo¬ 
nent of the loop series ( |39F associated with the loop type 
(xo, Xc, y). In order to prove Lemma|4] we have to find rj < 0 
such that is non-positive on D{p) x [ 77 ,-|-cx)[. 

We first show that for any rji < 0, there exists a 
neighborhood U of {xo,Xc,y) = (0,0,0) such that 
is non-positive on U DDjp) x [rji, -|-oo[. For a fixed pi < 0 
we construct a function that is an upper bound of 
We restrict ourselves to the domain V H D{p) x [rji, -|-oo[, 
where V = B(0, l/Sr) is the ball of radius l/3r centered at 
( 0 , 0 , 0 ). 

Let us explicitly write down the function (1581) term by 
term 














{xo,Xc,y,r]) = -2g{xo{l- p) + Xcp) 

+ {pxc - (1 - p) Xq) In 


1 -P 
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I / 

-lh2iil- p) Xo+pXc) 

+ (1 - p) /l 2 (xo) + ph 2 (Xc) 
I 
r 



We bound each term of (I59l l separately. Denote the fraction 
of variable nodes in the loop by X = xo (1 — p) + Xcp. The 
inequalities below trivially hold 

(pXc - (1 - p) Xo) In f < 2lnf - X 

, L^/2J 




t=l 


2t J - ^^'^(2[r/2j 


-2r7(xo (1 - p) + Xcp) < -2piX. 


(60) 


As the entropy is a concave function, we have the following 
inequality 

(1 - p)/l2 (Xo) + p/i2 (Xc) < (61) 

Concativty of —xlnx gives us 


Lr/2J 


'Yrl2\ 


- ^ t/tlnpt < - X! 2 /* 
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Lr/2J 

W 21 S 
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(lrj2\ \ /lr/2\ 

- 

+rln([r/2j)X. (62) 

Note that since the domain is restricted to types in a ball 
of radius l/3r, the fraction of variable nodes in a loop is 
upper-bounded X < l/3r. In particular it implies that 

L^/2J /lr/2\ 

E t I ^^ 2 t 

^ 9 




= 

2 

1 

< - 

- 6 
1 

< 

e 


(63) 


where e is the Euler constant. Finally as the entropy is 


increasing on [O, i], we have 


< h.GA-), 


(64) 


The upper bound on the function ( l59b is simply the sum of 
Inequalities ( l60l l, (IhTT i. ( l62l i and depends only on the fraction 
of variable nodes in a loop i.e. (xo, Xc, y, g) = g^^^{X) 
and 

= ^2 i^-x) - {I- l)h2 (X) + MX,(65) 

where M is a constant independent of g and p 

M = 2ln[^\+nnf ) + Hn (Lr/2J) - 2pi. 

^ (66) 

Notice that = 0 and that the derivative 

behaves like (| — l) InX in the neighborhood of 0. Hence, 
for I > 3, there exists 5 > 0 such that g^^^ is negative 
on ]0,(5]. Therefore for all types {xo,Xc,y) £ D{p) in the 
domain U = B(0, (5) fl B(0, l/3r) and for all g G [pi, +oo[ 
we have 


p«^C(xo,x„y,p) < r^^(X) 
< 0 . 


(67) 


By hypothesis the maximum of (l45l l is uniquely achieved 
in (0, 0,0) for p = 0. It implies that there exists A < 0 such 
that 

max / (xo,Xc,p,p)+fc (xo,Xc,p,p) = A. (68) 

(xo,Xc,y)GD(p)\U 

Therefore for r] > 772 = A/2 

max g^^^ (xq, Xc,y,g) < A - 2p2 = 0. (69) 

(xo,Xc,y)&D(p)\U 

We see that p = max(pi,P 2 ) < 0 satisfies by construction 
the condition of Lemma 01 □ 

Appendix II 
Proof of Lemma|5] 

Proof. We prove Lemma |5] only for the BSC (the proof for 
the BEC is almost identical). For a given p and p, we recall 


the function g^^p = and = p®®*^ as defined in 


BSC — BSC 
yp,p — y 

Appendix U We prove that for n sufficiently large and for 
all (5 £ Inn, ^/n~^\a.n , the function p® ^ is still 


p,p+s 

non-positive on D{p). 

First notice that the upper bound p®^^ does not de¬ 
pend on p. Using the same argument as in Appendix H] 
there exists a neighborhood U of (0,0,0) such that for 
all type (xo,Xc,p) G U H D{p + 6) and for all S £ 
n~^ Inn, Inn 


Pp.p+5 (xo, Xc, p, 0) < p®^^'^ (X) 

< 0 . 


( 70 ) 


BSC 


It remains to show that the variation of g 


on 















D {p + 5) \ U is bounded. Let us make the change of 
variables {xo,Xc) -)> {X,Xc) and {xo,Xc,y,0) 

9p^p+s {X,Xc,y,0). The following inequality holds 



+g^pf{X,x,,y,0). (71) 


Hence we can bound the maximum of g^^^ on D{p + S) \ U 
by 


max iX,Xc,yjO) < 

{X,x,,y)eDip+S)\U ^ c,y, ) _ 


max 

{X,Xc,y)&D{p)\U 


9p^p ^c,y, O) +c^^^(12) 


The maximum of g^^^ (^X, Xc,y,0) on D{p + S) \ U 
is by hypothesis negative (see Equation (|68]l)- There¬ 
fore for n sufficiently large we have that for all S G 

—v/n~^ Inn, v/n~^ Inn 


max (xo,Xa,y,0) < 0, (73) 

(xo,Xc,y)eD(p+6)\U — 


which concludes the proof. 
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